NON-DISCRETE COMPLEX HYPERBOLIC TRIANGLE GROUPS 

OF TYPE (m,m, cx)) 



ANNA PRATOUSSEVITCH 



Abstract. In this note we prove that a complex hyperbohc triangle group of 
type (m, m, oo), i.e. a group of isometries of the complex hyperbolic plane, 
generated by complex reflections in three complex geodesies meeting at angles 
7r/m, 7r/m and 0, is not discrete if the product of the three generators is regular 
elliptic. 



1. Introduction 

We study representations of real hyperbolic triangle groups, i.e. groups generated 
by reflections in the sides of triangles in H^, in the holomorphic isometry group 
PU(2, 1) of the complex hyperbolic plane H^. 

For the basic notions of complex hyperbolic geometry, especially for the complex 
hyperbolic plane H^, see for example section 2 in [Pra05]. The general references 
on complex hyperbolic geometry are [Gol99, Par03]. 

We use the following terminology: A complex hyperbolic triangle is a triple 
(Ci, C2, C3) of complex geodesies in H^. If the complex geodesies Ck-i and Ck+i 
meet at the angle n/pk we call the triangle (Ci,C2,C3) a {pi,p2,P3)-triangle. 

We call a subgroup of PU(2, 1) generated by complex reflections Lk in the sides Ck 
of a complex hyperbolic (pi,p2,P3)-triangle (Ci, C2, C3) a {pi,p2,P3) -triangle group. 
A {pi,p2,P3)-representation is a representation of the group 

r(pi,P2,P3) = (7i;72,73 I 7fc = ilk-Uk+iT" = 1 for aU k e {1,2,3}), 

where 7^+3 = 7fc, and the relation {■jk-ilk+iY'' = 1 is to omit for pk = 00, 
into the group PU(2,1), given by taking the generators jk of T{pi,p2,P3) to the 
generators Lk of a (pi,p2,P3)-triangle group. 

We prove in this paper the following result: 

Theorem. An {m,m, 00) -triangle group is not discrete if the product of the three 
generators is regular elliptic. 

More about the recent developments in the study of non-discrete complex hy- 
perbolic triangle groups of type {m,m, 00) can be found in [Kani07], [KPT09]. 

Part of this work was done during the stay at IHES in spring 2004. I am grateful 
to IHES for its hospitality and support. I would like to thank Martin Deraux and 
Richard Schwartz for useful conversations related to this work. 
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2. Non-Discreteness Proof 

For fixed {pi,P2,P7i) the space of complex hyperbolic triangle groups is of real 
dimension one. Wo now describe a parameterisation of the space of complex hyper- 
bolic triangles in iJ^ by means of an angular invariant a. See section 3 in [Pra05] 
for details. 

Let Cfc be the normalised polar vector of the complex geodesic Cfc. Let rk = 
|(c;s_i, Cfe+i)|. If the complex geodesies Ck-i and Ck+i meet at the angle ipk, then 
rk = cos ifk- We define the angular invariant a of the triangle (Ci,C2,C3) by 

a = arg ^ JJ(cfe_i, Cfc+i> j . 

A complex hyperbolic triangle in is determined uniquely up to isometry by the 
three angles and the angular invariant a (compare proposition 1 in [Pra05]). Let 
t'k = i^Ck be the complex reflection in the complex geodesic Cfe. 

Let (f> : T{pi,p2,P3) —5- PU(2, 1) be a complex hyperbolic triangle group repre- 
sentation and G := 4'(^ipiTP2,P3)) the corresponding complex hyperbolic triangle 
group. Assume that 7 is an element of infinite order in I'{pi,P2,P3) and that its 
image (('{'j) in G is regular elliptic. Then there are two cases, either (/)(7) is of finite 
order, then (j) is not injective, or 4>{'j) is of infinite order, then (j) is not discrete 
because the subgroup of G generated by ^(7) is not discrete. 

We shall show, that if the element tii2'-3 is regular elliptic, then it is not of finite 
order, hence the corresponding triangle group is not discrete. 

This statement was proved in [SchOl] for ideal triangle groups, i.e. groups of type 
(00, 00, 00). The statement for (m, m, cx))-triangle groups was formulated in [WGOO] 
(Lemma 3.4.0.19), but the proof there had a gap. 

Theorem. An {m,m, 00) -triangle group is not discrete if the product of the three 
generators is regular elliptic. 

Proof. We assume that the element L1L2L3 is regular elliptic of finite order. Let r 
— 1 be the trace of the corresponding matrix in SU(2, 1). The eigenvalues of this 
matrix are then three roots of unity with product equal to 1. Hence 

for some fci, k2, and with ^1+^2 + ^3 = 0. Here a;„ = cxp(27r7'/n) and n is 
taken as small as possible. On the other hand, the trace r can be computed (see 
section 8 in [Pra05]) as 

T = Snrarse'" - (4(rf + rf + r^) - 3). 

Let 

r = cos ( — I . 

vrn/ 

For (to, TO, oo)-groups we have n = r2 = r = cos(7r/TO) and rs = 1, hence 

r = (8r2)e*" - (Sr^ + 1). 

This equation implies that the complex number r 7^ — 1 lies on the circle with center 
in — (8r^ + 1) and radius 8r^, or in other words r satisfies the equation 

(r + (8r' + 1)) • (f + (8r' + 1)) = |t + (8r' + 1)|' = (8r')'. 



NON-DISCRETE COMPLEX HYPERBOLIC TRIANGLE GROUPS OF TYPE (m, m, oo) 3 



This implies in particular 

Re(r) < -1. 



Let N be the least common multiple of n and 2m. Let ak be the homomorphism 

of Q[ojn] given by (Jk{ujN) = F 
to Q[a;„] is a Galois automorphism. 



of Q[(xi7v] given by Gk{ujN) = u^. For k relatively prime to n the restriction of Uk 



Lemma 1. Let r = + lo^ + uj'^ he the trace of the matrix of lxL2L^, where n 
is taken as small as possible. Then at (r) satisfies the equation 

Wkir) + ak{8r^) + 1\ = ak{8r^). 

This implies in particular 

Re(afc(T)) ^ -1. 

Proof. We have 

r e Q[w„] C Q[ujn] 

and 

2r = 2cos (^^^ = a;2m + i^2m e Q[w2to] C Q[a;jv], 

hence the equation |r+ (8r^ + l)| = 8r^ is defined in Q[wAr]. The homomorphism cr/j 
commuters with complex conjugation and hence maps real numbers to real numbers. 
Applying the homomorphism ak to the equation 

(r + (8r2 + 1)) • (f + (8r2 + 1)) = {8r^f 

we obtain 

{<Tk{r) + ak{8r^ + l))(afc(f ) + ak{8r'' + 1)) = {ak{8r'')f. 

This equation is equivalent to 

kfe(T) + (7fe(8r2) + 1|2 = (afc(8r2))2. 

Since ak{'2r) is a real number, the number c7fe(8r^) = 2(cr/s(2r))^ is a non-negative 
real number. Hence akir) satisfies the equation 

\ak{T)+ak{8r'^) + l\=ak{8r'^). 

This equation means that the complex number akir) lies on the circle with center 
in — (fTfe(8r^) + 1) < and radius a-fe(8r^) ^ 0. This implies in particular 

ReK(T)) < -1. □ 

Lemma 2. Let r = cj^^ + lo^ + uj^ be the trace of the matrix of i\L2ii, where n 
is taken as small as possible. For i e {1, 2, 3}, let 

di = 



gcd(fci,n)' 

where gcd is the greatest common divisor. Then 

1 1 1 



ip{di) ip{d2) ipids) 
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Proof. According to Lemma 1, 



ReK(T)) s$ -1 



for any homomorphism a^- Summing over allfcG{l,...,n — 1} relatively prime 
to n we obtain 



, (fc,n) = l 



and hence 



((=,„) = ! 



> iy?(n). 



Here (/5 is the Euler (/j-function. The root of unity w^' is a primitive dj-th root of 
unity. The sum of all dj-th primitive roots of unity is in {—1,0, 1}, and hence is 
bounded by 1. The map (Z/nZ)* (Z/d^Z)* induced by the map Z/nZ ^ Z/d,Z 
is surjective, and the preimage of any element in (Z/djZ)* consists of (fi{n)/(fi{di) 
elements. Hence we obtain the inequality 



for i € {1, 2, 3}. From the inequalities 



(fc,n) = l 



(fc,„) = l 



ip{di) ip{d2) ifids) 



■(p{n) 



it follows 



111. 

+ -TTT + > 1- 



ip{di) ip{d2) ifids) 



□ 



The inequality 



111. 

+ ^TT + > 1 



ip{di) ' ip{d2) ' ip{d:i) 

implies that the triple {(fi{di) , ip{d2) , (pid^)) is equal (up to permutation) to one of 
the triples 

(1,?,?), (2,2,?), (2,3,3), (2,3,4), (2,3,5). 

But for the Eulcr (^-function we have <fi(x) = 1 for x € {1,2}, (p{x) = 2 for 
X E {3, 4, 6} and (p{x) ^ 4 for all other positive integers x. 

• The triples (2,3,3), (2,3,4), (2,3,5) cannot occur since ip{x) ^ 3 for any inte- 
ger X. 
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• Let 'p{di) = 1 for some i £ {1,2,3}. Without loss of generality we can as- 
sume that '^{di) = 1. Then di £ {1,2}, therefore {ki,n) G {n/2,n} and fci = 
0,n/2mod n. Hence uj^^ e {1,-1}. If /ci = then k2 + = 0. Let k = k2, 
then ka = —k and 

r = + + = 1 + + c.-'^ = 1 + 2 cos(2^fc/n) 

and Re(r) = 1 + 2cos(27rfc/n) > —1 in contradiction to Re(T) < —1. If fci = n/2 
then ^2 + ^3 = —n/2. Let fc = /;;2, then k^ = —k — n/2 and 

T = wf,! + wf," + wf;' = -l+w^-w^'' = -l + 2j sin(27rA;/n) 

and Re(T) = — 1 in contradiction to Re(r) < — 1. 

• If (p{di) — f{dj) = 2 for i,j S {1,2,3}, i ^ j, then di,dj S {3,4,6}, there- 
fore {ki,n) G {n/6,n/4, n/3} and /cj = ±n/6, ±n/4, ±n/3mod n. Hence w^* e 
{a^^, a^^, a^**}, where a = Wi2 = exp(27ri/12), and 

T = aP + a9+a'', p + g + r = 0, p, g G {±2, ±3, ±4}. 

Using Re(r) < —1 and Rc(q:'') ^ —1 wc obtain 

Re(af + a«) = Re(T) - Re(a'') < -1 + 1 = 0. 

Since Re(Q:^^) = |, Re(a^^) = and Re(a^'') = — |, we can only have Re(aP + 
a«) < if cyP + a" = a^^ + a^^ or + a? = a^^ + a^^. Out of these cases, 
we easily check that Re(aP + + a"^"^) < — 1 holds only if = a"^ = 
or = a'' = a~^, i.e. if r = 3a^^, but then a suitable homomorphism ak has 
the property Re(crfc(r)) > — 1 in contradiction to Lemma 1. 

□ 
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